Effect of Task on Time Spent Reading as an Implicit Measure of Interest

Information Filtering systems learn user preferences either through explicit or implicit feedback. However, requiring users to explicitly rate items as part of the interface interaction can place a large burden on the user. Implicit feedback removes the burden of explicit user ratings by transparently monitoring user behavior such as time spent reading, mouse movement and scrolling behavior. Previous research has shown that task may affect the effectiveness of some implicit measures. In this work we report both qualitative and quantitative results of an initial study examining the relationship between user time spent reading and relevance for three web search tasks: relevance judgment, simple question answering and complex question answering. This study indicates that the usefulness of time spent as a measure of user interest is related to task and becomes more useful as the task becomes more complex. Future directions for this research are presented. Introduction The rapid growth of the World Wide Web has highlighted the need for systems that help users sift through the wealth of information available online in order to find documents relevant to their needs. Information Filtering (IF) systems select documents for users based on their previous preferences. In order to learn user preferences, IF systems must determine user interest through either explicit or implicit user feedback. The next generation of IF systems must respond to the context of past user preferences as well as their current task, and it must do so without incurring additional overhead from the user. Explicit feedback may come in the form of user defined rules or more commonly as user ratings. Users are often required to give binary judgments after reading each document, such as “relevant” or “not relevant”, or rate items on a five point Likert scale. Implicit feedback transparently monitors user behavior in order to determine user preferences. Common user behaviors that have been found to indicate interest include time spent reading, mouse movements, scrolling behavior, and interactions with a document, such as saving, forwarding and printing. Implicit feedback has a number of advantages over explicit feedback. Unobtrusively monitoring user behavior allows users to focus on the task at hand without the interruption of having to give ratings for documents. Requiring users to stop and explicitly rate items before moving on to the next document disrupts a user’s normal reading and browsing behavior (Middleton et al., 2003; Nichols, 1997) and complicates the design. It is difficult to motivate users to continuously give explicit ratings (Kim et al., 2002; Konstan et al., 1997) even when the benefits are obvious (i.e., personalized recommendations). Implicit feedback may also help to eliminate inconsistencies in explicit user ratings. A hurdle for IF systems is user attitude towards having their behavior monitored. If users are not comfortable with having some behaviors monitored in order to make recommendations then an IF system with implicit feedback may be unproductive. IF systems can be used to make recommendations for users engaged in a variety of tasks, such as news reading, web browsing and web retrieval. Previous research has shown that the effectiveness of some implicit measures of interest may be dependent on the task at hand. For instance, time spent reading has been found to be a good indicator of interest for news reading (Konstan et al., 1997; Morita & Shinoda, 1994) and web browsing (Claypool et al., 2001; Seo & Zhang, 2000), but contradictory results have been found for Information Retrieval tasks (Kelly & Belkin, 2001; White et al., 2002). Based on this previous work, we have chosen to concentrate on evaluating the effectiveness of time spent reading for tasks related to web searching. We conducted a study examining the use of time spent reading as an implicit indicator of interest for three web search tasks: relevance judgment, simple question answering (fact finding), and complex question answering. We will present quantitative results from this study as well as qualitative results from user questionnaires exploring user attitudes towards IF systems that use implicit feedback. Following, we present previous findings with implicit measures and the affect of task on the effectiveness of implicit measures. We then present a detailed methodology of the study we performed followed by the results of the study. Lastly, we end with a discussion and future direction for this work.